2025-05-23 21:38:03,226 [ 608706 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-05-23 21:38:03,226 [ 608706 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2025-05-23 21:38:03,226 [ 608706 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2025-05-23 21:38:03,226 [ 608706 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_mumvsz --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel2_1.jsonl --report-log-exclude-logs-on-passed-tests test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_hosts_limit test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_s3_table_engine 'test_s3_cluster/test.py::test_hive_partitioning[0]' 'test_s3_cluster/test.py::test_hive_partitioning[1]' test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster -vvv " altinityinfra/integration-tests-runner:ad96270260ff '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [19 items] scheduling tests via LoadFileScheduling test_s3_cluster/test.py::test_cluster_default_expression test_reload_clusters_config/test.py::test_delete_cluster [gw0] [ 5%] FAILED test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection [gw0] [ 10%] FAILED test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_hosts_limit [gw0] [ 15%] FAILED test_s3_cluster/test.py::test_cluster_hosts_limit test_s3_cluster/test.py::test_cluster_with_header [gw0] [ 21%] FAILED test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection [gw0] [ 26%] FAILED test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count [gw0] [ 31%] FAILED test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro [gw0] [ 36%] FAILED test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_s3_table_engine [gw0] [ 42%] FAILED test_s3_cluster/test.py::test_distributed_s3_table_engine test_s3_cluster/test.py::test_hive_partitioning[0] [gw0] [ 47%] FAILED test_s3_cluster/test.py::test_hive_partitioning[0] test_s3_cluster/test.py::test_hive_partitioning[1] [gw0] [ 52%] FAILED test_s3_cluster/test.py::test_hive_partitioning[1] test_s3_cluster/test.py::test_remote_hedged [gw0] [ 57%] FAILED test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged [gw0] [ 63%] FAILED test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all [gw0] [ 68%] FAILED test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards [gw0] [ 73%] FAILED test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all [gw0] [ 78%] FAILED test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards [gw0] [ 84%] FAILED test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster [gw0] [ 89%] FAILED test_s3_cluster/test.py::test_wrong_cluster [gw1] [ 94%] PASSED test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster [gw1] [100%] FAILED test_reload_clusters_config/test.py::test_update_one_cluster =================================== FAILURES =================================== _______________________ test_cluster_default_expression ________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_default_expression(started_cluster): node = started_cluster.instances["s0_0_0"] node.query( f"insert into function s3('http://minio1:9001/root/data/data1', 'minio', '{minio_secret_key}', JSONEachRow) select 1 as id settings s3_truncate_on_insert=1" ) node.query( f"insert into function s3('http://minio1:9001/root/data/data2', 'minio', '{minio_secret_key}', JSONEachRow) select * from numbers(0) settings s3_truncate_on_insert=1" ) node.query( f"insert into function s3('http://minio1:9001/root/data/data3', 'minio', '{minio_secret_key}', JSONEachRow) select 2 as id settings s3_truncate_on_insert=1" ) expected_result = node.query( f"SELECT * FROM s3('http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id" ) result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id" ) assert result == expected_result result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'auto', 'id UInt32, date Date DEFAULT 18262') order by id" ) assert result == expected_result result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id" ) assert result == expected_result result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'auto', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id" ) assert result == expected_result result = node.query( "SELECT * FROM s3Cluster(cluster_simple, test_s3_with_default) order by id" ) assert result == expected_result > result = node.query( f"""SELECT * FROM s3('http://minio1:9001/root/data/data{{1,2,3}}', 'minio', '{minio_secret_key}', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:635: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml data/clickhouse/part1.csv data/clickhouse/part123.csv data/database/part2.csv data/database/partition675.csv data/generated/file_0.csv data/generated/file_1.csv data/generated/file_10.csv data/generated/file_11.csv data/generated/file_12.csv data/generated/file_13.csv data/generated/file_14.csv data/generated/file_15.csv data/generated/file_16.csv data/generated/file_17.csv data/generated/file_18.csv data/generated/file_19.csv data/generated/file_2.csv data/generated/file_20.csv data/generated/file_21.csv data/generated/file_22.csv data/generated/file_23.csv data/generated/file_24.csv data/generated/file_25.csv data/generated/file_26.csv data/generated/file_27.csv data/generated/file_28.csv data/generated/file_29.csv data/generated/file_3.csv data/generated/file_30.csv data/generated/file_31.csv data/generated/file_32.csv data/generated/file_33.csv data/generated/file_34.csv data/generated/file_35.csv data/generated/file_36.csv data/generated/file_37.csv data/generated/file_38.csv data/generated/file_39.csv data/generated/file_4.csv data/generated/file_40.csv data/generated/file_41.csv data/generated/file_42.csv data/generated/file_43.csv data/generated/file_44.csv data/generated/file_45.csv data/generated/file_46.csv data/generated/file_47.csv data/generated/file_48.csv data/generated/file_49.csv data/generated/file_5.csv data/generated/file_50.csv data/generated/file_51.csv data/generated/file_52.csv data/generated/file_53.csv data/generated/file_54.csv data/generated/file_55.csv data/generated/file_56.csv data/generated/file_57.csv data/generated/file_58.csv data/generated/file_59.csv data/generated/file_6.csv data/generated/file_60.csv data/generated/file_61.csv data/generated/file_62.csv data/generated/file_63.csv data/generated/file_64.csv data/generated/file_65.csv data/generated/file_66.csv data/generated/file_67.csv data/generated/file_68.csv data/generated/file_69.csv data/generated/file_7.csv data/generated/file_70.csv data/generated/file_71.csv data/generated/file_72.csv data/generated/file_73.csv data/generated/file_74.csv data/generated/file_75.csv data/generated/file_76.csv data/generated/file_77.csv data/generated/file_78.csv data/generated/file_79.csv data/generated/file_8.csv data/generated/file_80.csv data/generated/file_81.csv data/generated/file_82.csv data/generated/file_83.csv data/generated/file_84.csv data/generated/file_85.csv data/generated/file_86.csv data/generated/file_87.csv data/generated/file_88.csv data/generated/file_89.csv data/generated/file_9.csv data/generated/file_90.csv data/generated/file_91.csv data/generated/file_92.csv data/generated/file_93.csv data/generated/file_94.csv data/generated/file_95.csv data/generated/file_96.csv data/generated/file_97.csv data/generated/file_98.csv data/generated/file_99.csv ---------------------------- Captured stderr setup ----------------------------- Command:[docker ps | wc -l] Stdout:1 No running containers Pruning Docker networks Command:[docker network prune --force] Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] Stdout:net.ipv4.ip_local_port_range = 55000 65535 ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 ENV HOSTNAME 4874f67d8af0 ENV SHLVL 0 ENV HOME /root ENV OLDPWD / ENV DOCKER_HELPER_TAG 5dc43a6382f0 ENV PYTHONUNBUFFERED 1 ENV DOCKER_PYTHON_BOTTLE_TAG d862517635bf ENV UBSAN_OPTIONS print_stacktrace=1 ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel2_1.jsonl --report-log-exclude-logs-on-passed-tests test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_hosts_limit test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_s3_table_engine 'test_s3_cluster/test.py::test_hive_partitioning[0]' 'test_s3_cluster/test.py::test_hive_partitioning[1]' test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster -vvv ENV COMPOSE_HTTP_TIMEOUT 600 ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ENV DOCKER_KERBERIZED_HADOOP_TAG latest ENV DOCKER_CHANNEL stable ENV DOCKER_CLIENT_TIMEOUT 300 ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e ENV PWD /ClickHouse/tests/integration ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config ENV TZ Etc/UTC ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java ENV DOCKER_BASE_TAG 5ccda723c1fc ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 ENV LC_CTYPE C.UTF-8 ENV INTEGRATION_TESTS_RUN_ID 1 ENV WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 ENV PYTEST_XDIST_TESTRUNUID 6ee8473ea4724730b756ab18284585f7 ENV PYTEST_XDIST_WORKER gw0 ENV PYTEST_XDIST_WORKER_COUNT 10 ENV PYTEST_CURRENT_TEST test_s3_cluster/test.py::test_cluster_default_expression (setup) CLUSTER INIT base_config_dir:/clickhouse-config clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Setup Keeper Cluster name: project_name:roottests3cluster-gw0. Added instance name:s0_0_0 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env', '--project-name', 'roottests3cluster-gw0', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottests3cluster-gw0. Added instance name:s0_0_1 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env', '--project-name', 'roottests3cluster-gw0', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log Cluster name: project_name:roottests3cluster-gw0. Added instance name:s0_1_0 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env', '--project-name', 'roottests3cluster-gw0', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ Starting cluster... Running tests in /ClickHouse/tests/integration/test_s3_cluster/test.py Cluster start called. is_up=False Docker networks for project roottests3cluster-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster-gw0 are DRIVER VOLUME NAME Cleanup called Docker networks for project roottests3cluster-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3cluster-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottests3cluster-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 Setup directory for instance: s0_0_0 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/configs/config.d Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/database Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: s0_0_1 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/configs/config.d Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/database Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Setup directory for instance: s0_1_0 Create directory for configuration generated in this helper Create directory for common tests configuration Copy common configuration from helpers Generate and write macros file Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/configs/config.d Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/database Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/logs Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/coordination', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] No config file found http://localhost:None "GET /version HTTP/1.1" 200 826 Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml pull] Stderr: s0_1_0 Skipped - Image is already being pulled by zoo2 Stderr: s0_0_1 Skipped - Image is already being pulled by zoo2 Stderr: zoo1 Skipped - Image is already being pulled by zoo2 Stderr: s0_0_0 Skipped - Image is already being pulled by zoo2 Stderr: zoo3 Skipped - Image is already being pulled by zoo2 Stderr: proxy1 Skipped - Image is already being pulled by proxy2 Stderr: zoo2 Pulling Stderr: minio1 Pulling Stderr: proxy2 Pulling Stderr: resolver Pulling Stderr: resolver Pulled Stderr: minio1 Pulled Stderr: zoo2 Pulled Stderr: proxy2 Pulled Setup ZooKeeper Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/coordination'] Command:[docker compose --project-name roottests3cluster-gw0 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] Stderr:time="2025-05-23T21:38:20Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Network roottests3cluster-gw0_default Creating Stderr: Network roottests3cluster-gw0_default Created Stderr: Container roottests3cluster-gw0-zoo1-1 Creating Stderr: Container roottests3cluster-gw0-zoo2-1 Creating Stderr: Container roottests3cluster-gw0-zoo3-1 Creating Stderr: Container roottests3cluster-gw0-zoo3-1 Created Stderr: Container roottests3cluster-gw0-zoo2-1 Created Stderr: Container roottests3cluster-gw0-zoo1-1 Created Stderr: Container roottests3cluster-gw0-zoo1-1 Starting Stderr: Container roottests3cluster-gw0-zoo2-1 Starting Stderr: Container roottests3cluster-gw0-zoo3-1 Starting Stderr: Container roottests3cluster-gw0-zoo3-1 Started Stderr: Container roottests3cluster-gw0-zoo2-1 Started Stderr: Container roottests3cluster-gw0-zoo1-1 Started Stderr:time="2025-05-23T21:38:21Z" level=debug msg="otel error" error="" Stderr:time="2025-05-23T21:38:21Z" level=debug msg="otel error" error="" Wait ZooKeeper to start get_instance_ip instance_name=zoo1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-zoo1-1/json HTTP/1.1" 200 None get_kazoo_client: zoo1, ip:172.16.2.3, port:2181, use_ssl:False Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Connection dropped: socket connection error: Connection refused Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo2 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-zoo2-1/json HTTP/1.1" 200 None get_kazoo_client: zoo2, ip:172.16.2.4, port:2181, use_ssl:False Connecting to 172.16.2.4(172.16.2.4):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED get_instance_ip instance_name=zoo3 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-zoo3-1/json HTTP/1.1" 200 None get_kazoo_client: zoo3, ip:172.16.2.2, port:2181, use_ssl:False Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) Zookeeper connection established, state: CONNECTED Sending request(xid=1): GetChildren(path='/', watcher=None) Received response(xid=1): ['keeper'] Sending request(xid=2): Close() Connection dropped: socket connection broken Transition to CONNECTING Zookeeper connection lost Failed connecting to Zookeeper within the connection retry policy. Zookeeper session closed, state: CLOSED All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') Trying to create Minio instance by command docker compose --project-name roottests3cluster-gw0 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d Command:[docker compose --project-name roottests3cluster-gw0 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] Stderr:time="2025-05-23T21:38:24Z" level=trace msg="Docker Desktop integration not enabled" Stderr: Volume "roottests3cluster-gw0_data1-1" Creating Stderr: Volume "roottests3cluster-gw0_data1-1" Created Stderr:time="2025-05-23T21:38:24Z" level=warning msg="Found orphan containers ([roottests3cluster-gw0-zoo3-1 roottests3cluster-gw0-zoo2-1 roottests3cluster-gw0-zoo1-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." Stderr: Container roottests3cluster-gw0-proxy2-1 Creating Stderr: Container roottests3cluster-gw0-proxy1-1 Creating Stderr: Container roottests3cluster-gw0-proxy1-1 Created Stderr: Container roottests3cluster-gw0-proxy2-1 Created Stderr: Container roottests3cluster-gw0-minio1-1 Creating Stderr: Container roottests3cluster-gw0-resolver-1 Creating Stderr: Container roottests3cluster-gw0-resolver-1 Created Stderr: Container roottests3cluster-gw0-minio1-1 Created Stderr: Container roottests3cluster-gw0-proxy2-1 Starting Stderr: Container roottests3cluster-gw0-proxy1-1 Starting Stderr: Container roottests3cluster-gw0-proxy1-1 Started Stderr: Container roottests3cluster-gw0-proxy2-1 Started Stderr: Container roottests3cluster-gw0-resolver-1 Starting Stderr: Container roottests3cluster-gw0-minio1-1 Starting Stderr: Container roottests3cluster-gw0-resolver-1 Started Stderr: Container roottests3cluster-gw0-minio1-1 Started Stderr:time="2025-05-23T21:38:25Z" level=debug msg="otel error" error="" Stderr:time="2025-05-23T21:38:25Z" level=debug msg="otel error" error="" Trying to connect to Minio... get_instance_ip instance_name=minio1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-minio1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=proxy1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-proxy1-1/json HTTP/1.1" 200 None Starting new HTTP connection (1): 172.16.2.8:9001 Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (2): 172.16.2.8:9001 Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (3): 172.16.2.8:9001 Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / Starting new HTTP connection (4): 172.16.2.8:9001 Can't connect to Minio: HTTPConnectionPool(host='172.16.2.8', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) Starting new HTTP connection (5): 172.16.2.8:9001 http://172.16.2.8:9001 "GET / HTTP/1.1" 200 0 Connected to Minio. http://172.16.2.8:9001 "GET /root?location= HTTP/1.1" 404 0 http://172.16.2.8:9001 "PUT /root HTTP/1.1" 200 0 S3 bucket 'root' created http://172.16.2.8:9001 "GET /root2?location= HTTP/1.1" 404 0 http://172.16.2.8:9001 "PUT /root2 HTTP/1.1" 200 0 S3 bucket 'root2' created ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml up -d --no-recreate') Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml up -d --no-recreate] Stderr: Container roottests3cluster-gw0-proxy1-1 Running Stderr: Container roottests3cluster-gw0-proxy2-1 Running Stderr: Container roottests3cluster-gw0-resolver-1 Running Stderr: Container roottests3cluster-gw0-zoo2-1 Running Stderr: Container roottests3cluster-gw0-zoo1-1 Running Stderr: Container roottests3cluster-gw0-zoo3-1 Running Stderr: Container roottests3cluster-gw0-minio1-1 Running Stderr: Container roottests3cluster-gw0-s0_0_0-1 Creating Stderr: Container roottests3cluster-gw0-s0_0_1-1 Creating Stderr: Container roottests3cluster-gw0-s0_1_0-1 Creating Stderr: Container roottests3cluster-gw0-s0_1_0-1 Created Stderr: Container roottests3cluster-gw0-s0_0_1-1 Created Stderr: Container roottests3cluster-gw0-s0_0_0-1 Created Stderr: Container roottests3cluster-gw0-s0_0_1-1 Starting Stderr: Container roottests3cluster-gw0-s0_1_0-1 Starting Stderr: Container roottests3cluster-gw0-s0_0_0-1 Starting Stderr: Container roottests3cluster-gw0-s0_0_1-1 Started Stderr: Container roottests3cluster-gw0-s0_0_0-1 Started Stderr: Container roottests3cluster-gw0-s0_1_0-1 Started ClickHouse instance created get_instance_ip instance_name=s0_0_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_0-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_0_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_0-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_0_0, ip: 172.16.2.10... http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None ClickHouse s0_0_0 started get_instance_ip instance_name=s0_0_1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_1-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_0_1 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_1-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_0_1, ip: 172.16.2.11... http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_1-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/e3039622a050546626d8f247a1e004428b0236bfb8dca3e1a7511b2e3fbde01a/json HTTP/1.1" 200 None ClickHouse s0_0_1 started get_instance_ip instance_name=s0_1_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_1_0-1/json HTTP/1.1" 200 None get_instance_ip instance_name=s0_1_0 http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_1_0-1/json HTTP/1.1" 200 None Waiting for ClickHouse start in s0_1_0, ip: 172.16.2.9... http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_1_0-1/json HTTP/1.1" 200 None http://localhost:None "GET /v1.46/containers/ac045d7153b9cd0642b097e41fdf278e1b431bc473ecebf7f57cae2b779f0ac8/json HTTP/1.1" 200 None ClickHouse s0_1_0 started Cluster started http://172.16.2.8:9001 "PUT /root/data/clickhouse/part1.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/clickhouse/part123.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/database/part2.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/database/partition675.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_0.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_1.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_2.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_3.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_4.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_5.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_6.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_7.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_8.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_9.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_10.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_11.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_12.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_13.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_14.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_15.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_16.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_17.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_18.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_19.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_20.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_21.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_22.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_23.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_24.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_25.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_26.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_27.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_28.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_29.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_30.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_31.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_32.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_33.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_34.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_35.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_36.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_37.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_38.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_39.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_40.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_41.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_42.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_43.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_44.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_45.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_46.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_47.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_48.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_49.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_50.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_51.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_52.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_53.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_54.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_55.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_56.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_57.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_58.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_59.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_60.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_61.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_62.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_63.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_64.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_65.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_66.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_67.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_68.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_69.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_70.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_71.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_72.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_73.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_74.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_75.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_76.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_77.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_78.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_79.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_80.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_81.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_82.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_83.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_84.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_85.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_86.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_87.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_88.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_89.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_90.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_91.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_92.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_93.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_94.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_95.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_96.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_97.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_98.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "PUT /root/data/generated/file_99.csv HTTP/1.1" 200 0 http://172.16.2.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1" 200 0 Starting mock server s3_mock.py run container_id:roottests3cluster-gw0-resolver-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname s3_mock.py) && echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py'] Command:[docker exec roottests3cluster-gw0-resolver-1 bash -c mkdir -p $(dirname s3_mock.py) && echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py] run container_id:roottests3cluster-gw0-resolver-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log'] Command:[docker exec roottests3cluster-gw0-resolver-1 bash -c python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log] run container_id:roottests3cluster-gw0-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] Command:[docker exec roottests3cluster-gw0-resolver-1 curl -s http://localhost:8080/] Exitcode:7 run container_id:roottests3cluster-gw0-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] Command:[docker exec roottests3cluster-gw0-resolver-1 curl -s http://localhost:8080/] Stdout:OK s3_mock.py answered OK on attempt 2 Mock server s3_mock.py started ------------------------------ Captured log setup ------------------------------ 2025-05-23 21:38:09 [ 684 ] DEBUG : Command:[docker ps | wc -l] (cluster.py:121, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : No running containers (conftest.py:95, cleanup_environment) 2025-05-23 21:38:09 [ 684 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment) 2025-05-23 21:38:09 [ 684 ] DEBUG : Command:[docker network prune --force] (cluster.py:121, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] (cluster.py:121, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:145, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV HOSTNAME 4874f67d8af0 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV SHLVL 0 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV HOME /root (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV OLDPWD / (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_HELPER_TAG 5dc43a6382f0 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG d862517635bf (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel2_1.jsonl --report-log-exclude-logs-on-passed-tests test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_hosts_limit test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_s3_table_engine 'test_s3_cluster/test.py::test_hive_partitioning[0]' 'test_s3_cluster/test.py::test_hive_partitioning[1]' test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster -vvv (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV TZ Etc/UTC (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV DOCKER_BASE_TAG 5ccda723c1fc (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 1 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV PYTEST_XDIST_TESTRUNUID 6ee8473ea4724730b756ab18284585f7 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV PYTEST_XDIST_WORKER gw0 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV PYTEST_XDIST_WORKER_COUNT 10 (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : ENV PYTEST_CURRENT_TEST test_s3_cluster/test.py::test_cluster_default_expression (setup) (cluster.py:419, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:719, __init__) 2025-05-23 21:38:09 [ 684 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1656, add_instance) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup Keeper (cluster.py:1021, setup_keeper_cmd) 2025-05-23 21:38:09 [ 684 ] DEBUG : Cluster name: project_name:roottests3cluster-gw0. Added instance name:s0_0_0 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env', '--project-name', 'roottests3cluster-gw0', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1942, add_instance) 2025-05-23 21:38:09 [ 684 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1656, add_instance) 2025-05-23 21:38:09 [ 684 ] DEBUG : Cluster name: project_name:roottests3cluster-gw0. Added instance name:s0_0_1 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env', '--project-name', 'roottests3cluster-gw0', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1942, add_instance) 2025-05-23 21:38:09 [ 684 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1656, add_instance) 2025-05-23 21:38:09 [ 684 ] DEBUG : Cluster name: project_name:roottests3cluster-gw0. Added instance name:s0_1_0 tag:5ccda723c1fc base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env', '--project-name', 'roottests3cluster-gw0', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1942, add_instance) 2025-05-23 21:38:09 [ 684 ] INFO : Starting cluster... (test.py:95, started_cluster) 2025-05-23 21:38:09 [ 684 ] INFO : Running tests in /ClickHouse/tests/integration/test_s3_cluster/test.py (cluster.py:2672, start) 2025-05-23 21:38:09 [ 684 ] DEBUG : Cluster start called. is_up=False (cluster.py:2679, start) 2025-05-23 21:38:09 [ 684 ] DEBUG : Docker networks for project roottests3cluster-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:825, print_all_docker_pieces) 2025-05-23 21:38:09 [ 684 ] DEBUG : Docker containers for project roottests3cluster-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:833, print_all_docker_pieces) 2025-05-23 21:38:09 [ 684 ] DEBUG : Docker volumes for project roottests3cluster-gw0 are DRIVER VOLUME NAME (cluster.py:841, print_all_docker_pieces) 2025-05-23 21:38:09 [ 684 ] DEBUG : Cleanup called (cluster.py:846, cleanup) 2025-05-23 21:38:09 [ 684 ] DEBUG : Docker networks for project roottests3cluster-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:825, print_all_docker_pieces) 2025-05-23 21:38:09 [ 684 ] DEBUG : Docker containers for project roottests3cluster-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:833, print_all_docker_pieces) 2025-05-23 21:38:09 [ 684 ] DEBUG : Docker volumes for project roottests3cluster-gw0 are DRIVER VOLUME NAME (cluster.py:841, print_all_docker_pieces) 2025-05-23 21:38:09 [ 684 ] DEBUG : Command:[docker container list --all --filter name='^/roottests3cluster-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : Unstopped containers: {} (cluster.py:860, cleanup) 2025-05-23 21:38:09 [ 684 ] DEBUG : No running containers for project: roottests3cluster-gw0 (cluster.py:874, cleanup) 2025-05-23 21:38:09 [ 684 ] DEBUG : Trying to prune unused networks... (cluster.py:880, cleanup) 2025-05-23 21:38:09 [ 684 ] DEBUG : Trying to prune unused images... (cluster.py:896, cleanup) 2025-05-23 21:38:09 [ 684 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : Images pruned (cluster.py:899, cleanup) 2025-05-23 21:38:09 [ 684 ] DEBUG : Trying to prune unused volumes... (cluster.py:905, cleanup) 2025-05-23 21:38:09 [ 684 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2025-05-23 21:38:09 [ 684 ] DEBUG : Volumes pruned: 1 (cluster.py:910, cleanup) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup directory for instance: s0_0_0 (cluster.py:2692, start) 2025-05-23 21:38:09 [ 684 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4536, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Create directory for common tests configuration (cluster.py:4541, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Copy common configuration from helpers (cluster.py:4561, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Generate and write macros file (cluster.py:4613, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/configs/config.d (cluster.py:4649, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/database (cluster.py:4666, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/logs (cluster.py:4677, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4758, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup directory for instance: s0_0_1 (cluster.py:2692, start) 2025-05-23 21:38:09 [ 684 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4536, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Create directory for common tests configuration (cluster.py:4541, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Copy common configuration from helpers (cluster.py:4561, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Generate and write macros file (cluster.py:4613, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/configs/config.d (cluster.py:4649, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/database (cluster.py:4666, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/logs (cluster.py:4677, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4758, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup directory for instance: s0_1_0 (cluster.py:2692, start) 2025-05-23 21:38:09 [ 684 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4536, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Create directory for common tests configuration (cluster.py:4541, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Copy common configuration from helpers (cluster.py:4561, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Generate and write macros file (cluster.py:4613, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_s3_cluster/configs/cluster.xml', '/ClickHouse/tests/integration/test_s3_cluster/configs/named_collections.xml'] to /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/configs/config.d (cluster.py:4649, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/database (cluster.py:4666, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/logs (cluster.py:4677, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4758, create_dir) 2025-05-23 21:38:09 [ 684 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/coordination', 'MINIO_CERTS_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/minio/certs', 'MINIO_DATA_DIR': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/minio/data', 'MINIO_PORT': '9001', 'SSL_CERT_FILE': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/minio/certs/public.crt', 'RESOLVER_LOGS': '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/resolver', 'RESOLVER_LOGS_FS': 'bind'} stored in /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env (cluster.py:96, _create_env_file) 2025-05-23 21:38:09 [ 684 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-05-23 21:38:09 [ 684 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-05-23 21:38:09 [ 684 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2025-05-23 21:38:09 [ 684 ] DEBUG : No config file found (config.py:28, find_config_file) 2025-05-23 21:38:09 [ 684 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2025-05-23 21:38:09 [ 684 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml pull] (cluster.py:121, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: s0_1_0 Skipped - Image is already being pulled by zoo2 (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: s0_0_1 Skipped - Image is already being pulled by zoo2 (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: zoo1 Skipped - Image is already being pulled by zoo2 (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: s0_0_0 Skipped - Image is already being pulled by zoo2 (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: zoo3 Skipped - Image is already being pulled by zoo2 (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: proxy1 Skipped - Image is already being pulled by proxy2 (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: zoo2 Pulling (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: minio1 Pulling (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: proxy2 Pulling (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: resolver Pulling (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: resolver Pulled (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: minio1 Pulled (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: zoo2 Pulled (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Stderr: proxy2 Pulled (cluster.py:147, run_and_check) 2025-05-23 21:38:20 [ 684 ] DEBUG : Setup ZooKeeper (cluster.py:2733, start) 2025-05-23 21:38:20 [ 684 ] DEBUG : Creating internal ZooKeeper dirs: ['/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper1/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper2/coordination', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/log', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/config', '/ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/keeper3/coordination'] (cluster.py:2734, start) 2025-05-23 21:38:20 [ 684 ] DEBUG : Command:[docker compose --project-name roottests3cluster-gw0 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --verbose up -d] (cluster.py:121, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr:time="2025-05-23T21:38:20Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Network roottests3cluster-gw0_default Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Network roottests3cluster-gw0_default Created (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr:time="2025-05-23T21:38:21Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Stderr:time="2025-05-23T21:38:21Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2025-05-23 21:38:21 [ 684 ] DEBUG : Wait ZooKeeper to start (cluster.py:2398, wait_zookeeper_to_start) 2025-05-23 21:38:21 [ 684 ] DEBUG : get_instance_ip instance_name=zoo1 (cluster.py:1999, get_instance_ip) 2025-05-23 21:38:21 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-zoo1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:21 [ 684 ] DEBUG : get_kazoo_client: zoo1, ip:172.16.2.3, port:2181, use_ssl:False (cluster.py:3234, get_kazoo_client) 2025-05-23 21:38:21 [ 684 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2025-05-23 21:38:21 [ 684 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-05-23 21:38:21 [ 684 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2025-05-23 21:38:21 [ 684 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-05-23 21:38:21 [ 684 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2025-05-23 21:38:21 [ 684 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-05-23 21:38:22 [ 684 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2025-05-23 21:38:22 [ 684 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-05-23 21:38:22 [ 684 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2025-05-23 21:38:22 [ 684 ] WARNING : Connection dropped: socket connection error: Connection refused (connection.py:622, _connect_attempt) 2025-05-23 21:38:24 [ 684 ] INFO : Connecting to 172.16.2.3(172.16.2.3):2181, use_ssl: False (connection.py:650, _connect) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-05-23 21:38:24 [ 684 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-05-23 21:38:24 [ 684 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-05-23 21:38:24 [ 684 ] DEBUG : get_instance_ip instance_name=zoo2 (cluster.py:1999, get_instance_ip) 2025-05-23 21:38:24 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-zoo2-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:24 [ 684 ] DEBUG : get_kazoo_client: zoo2, ip:172.16.2.4, port:2181, use_ssl:False (cluster.py:3234, get_kazoo_client) 2025-05-23 21:38:24 [ 684 ] INFO : Connecting to 172.16.2.4(172.16.2.4):2181, use_ssl: False (connection.py:650, _connect) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-05-23 21:38:24 [ 684 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-05-23 21:38:24 [ 684 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-05-23 21:38:24 [ 684 ] DEBUG : get_instance_ip instance_name=zoo3 (cluster.py:1999, get_instance_ip) 2025-05-23 21:38:24 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-zoo3-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:24 [ 684 ] DEBUG : get_kazoo_client: zoo3, ip:172.16.2.2, port:2181, use_ssl:False (cluster.py:3234, get_kazoo_client) 2025-05-23 21:38:24 [ 684 ] INFO : Connecting to 172.16.2.2(172.16.2.2):2181, use_ssl: False (connection.py:650, _connect) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=30000, session_id=0, passwd=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper connection established, state: CONNECTED (client.py:532, _session_callback) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=1): GetChildren(path='/', watcher=None) (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] DEBUG : Received response(xid=1): ['keeper'] (connection.py:410, _read_response) 2025-05-23 21:38:24 [ 684 ] DEBUG : Sending request(xid=2): Close() (connection.py:312, _submit) 2025-05-23 21:38:24 [ 684 ] WARNING : Connection dropped: socket connection broken (connection.py:622, _connect_attempt) 2025-05-23 21:38:24 [ 684 ] WARNING : Transition to CONNECTING (connection.py:626, _connect_attempt) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper connection lost (client.py:543, _session_callback) 2025-05-23 21:38:24 [ 684 ] WARNING : Failed connecting to Zookeeper within the connection retry policy. (connection.py:515, zk_loop) 2025-05-23 21:38:24 [ 684 ] INFO : Zookeeper session closed, state: CLOSED (client.py:537, _session_callback) 2025-05-23 21:38:24 [ 684 ] DEBUG : All instances of ZooKeeper started: ('zoo1', 'zoo2', 'zoo3') (cluster.py:2414, wait_zookeeper_nodes_to_start) 2025-05-23 21:38:24 [ 684 ] INFO : Trying to create Minio instance by command docker compose --project-name roottests3cluster-gw0 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d (cluster.py:2993, start) 2025-05-23 21:38:24 [ 684 ] DEBUG : Command:[docker compose --project-name roottests3cluster-gw0 --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --verbose up -d] (cluster.py:121, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr:time="2025-05-23T21:38:24Z" level=trace msg="Docker Desktop integration not enabled" (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Volume "roottests3cluster-gw0_data1-1" Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Volume "roottests3cluster-gw0_data1-1" Created (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr:time="2025-05-23T21:38:24Z" level=warning msg="Found orphan containers ([roottests3cluster-gw0-zoo3-1 roottests3cluster-gw0-zoo2-1 roottests3cluster-gw0-zoo1-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up." (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr:time="2025-05-23T21:38:25Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] DEBUG : Stderr:time="2025-05-23T21:38:25Z" level=debug msg="otel error" error="" (cluster.py:147, run_and_check) 2025-05-23 21:38:25 [ 684 ] INFO : Trying to connect to Minio... (cluster.py:2999, start) 2025-05-23 21:38:25 [ 684 ] DEBUG : get_instance_ip instance_name=minio1 (cluster.py:1999, get_instance_ip) 2025-05-23 21:38:25 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-minio1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:25 [ 684 ] DEBUG : get_instance_ip instance_name=proxy1 (cluster.py:1999, get_instance_ip) 2025-05-23 21:38:25 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-proxy1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:25 [ 684 ] DEBUG : Starting new HTTP connection (1): 172.16.2.8:9001 (connectionpool.py:245, _new_conn) 2025-05-23 21:38:25 [ 684 ] DEBUG : Incremented Retry for (url='/'): Retry(total=2, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-05-23 21:38:25 [ 684 ] WARNING : Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-05-23 21:38:25 [ 684 ] DEBUG : Starting new HTTP connection (2): 172.16.2.8:9001 (connectionpool.py:245, _new_conn) 2025-05-23 21:38:25 [ 684 ] DEBUG : Incremented Retry for (url='/'): Retry(total=1, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-05-23 21:38:25 [ 684 ] WARNING : Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-05-23 21:38:25 [ 684 ] DEBUG : Starting new HTTP connection (3): 172.16.2.8:9001 (connectionpool.py:245, _new_conn) 2025-05-23 21:38:25 [ 684 ] DEBUG : Incremented Retry for (url='/'): Retry(total=0, connect=None, read=None, redirect=None, status=None) (retry.py:517, increment) 2025-05-23 21:38:25 [ 684 ] WARNING : Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')': / (connectionpool.py:872, urlopen) 2025-05-23 21:38:25 [ 684 ] DEBUG : Starting new HTTP connection (4): 172.16.2.8:9001 (connectionpool.py:245, _new_conn) 2025-05-23 21:38:25 [ 684 ] DEBUG : Can't connect to Minio: HTTPConnectionPool(host='172.16.2.8', port=9001): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) (cluster.py:2516, wait_minio_to_start) 2025-05-23 21:38:26 [ 684 ] DEBUG : Starting new HTTP connection (5): 172.16.2.8:9001 (connectionpool.py:245, _new_conn) 2025-05-23 21:38:26 [ 684 ] DEBUG : http://172.16.2.8:9001 "GET / HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:26 [ 684 ] DEBUG : Connected to Minio. (cluster.py:2496, wait_minio_to_start) 2025-05-23 21:38:26 [ 684 ] DEBUG : http://172.16.2.8:9001 "GET /root?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:26 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:26 [ 684 ] DEBUG : S3 bucket 'root' created (cluster.py:2511, wait_minio_to_start) 2025-05-23 21:38:26 [ 684 ] DEBUG : http://172.16.2.8:9001 "GET /root2?location= HTTP/1.1" 404 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:26 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root2 HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:26 [ 684 ] DEBUG : S3 bucket 'root2' created (cluster.py:2511, wait_minio_to_start) 2025-05-23 21:38:26 [ 684 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml up -d --no-recreate') (cluster.py:3061, start) 2025-05-23 21:38:26 [ 684 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml up -d --no-recreate] (cluster.py:121, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Running (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Running (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Running (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Running (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Running (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Running (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Running (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Creating (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Created (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Starting (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Started (cluster.py:147, run_and_check) 2025-05-23 21:38:27 [ 684 ] DEBUG : ClickHouse instance created (cluster.py:3069, start) 2025-05-23 21:38:27 [ 684 ] DEBUG : get_instance_ip instance_name=s0_0_0 (cluster.py:1999, get_instance_ip) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : get_instance_ip instance_name=s0_0_0 (cluster.py:2009, get_instance_global_ipv6) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : Waiting for ClickHouse start in s0_0_0, ip: 172.16.2.10... (cluster.py:3077, start) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/e1f148aaf3da917ffa72487e9b4cc2ab301f3a79040232ede4b0ca68b96f627a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : ClickHouse s0_0_0 started (cluster.py:3081, start) 2025-05-23 21:38:27 [ 684 ] DEBUG : get_instance_ip instance_name=s0_0_1 (cluster.py:1999, get_instance_ip) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : get_instance_ip instance_name=s0_0_1 (cluster.py:2009, get_instance_global_ipv6) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : Waiting for ClickHouse start in s0_0_1, ip: 172.16.2.11... (cluster.py:3077, start) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_0_1-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/e3039622a050546626d8f247a1e004428b0236bfb8dca3e1a7511b2e3fbde01a/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : ClickHouse s0_0_1 started (cluster.py:3081, start) 2025-05-23 21:38:27 [ 684 ] DEBUG : get_instance_ip instance_name=s0_1_0 (cluster.py:1999, get_instance_ip) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_1_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : get_instance_ip instance_name=s0_1_0 (cluster.py:2009, get_instance_global_ipv6) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_1_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : Waiting for ClickHouse start in s0_1_0, ip: 172.16.2.9... (cluster.py:3077, start) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/roottests3cluster-gw0-s0_1_0-1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : http://localhost:None "GET /v1.46/containers/ac045d7153b9cd0642b097e41fdf278e1b431bc473ecebf7f57cae2b779f0ac8/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2025-05-23 21:38:27 [ 684 ] DEBUG : ClickHouse s0_1_0 started (cluster.py:3081, start) 2025-05-23 21:38:27 [ 684 ] INFO : Cluster started (test.py:97, started_cluster) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/clickhouse/part1.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/clickhouse/part123.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/database/part2.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/database/partition675.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_0.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_1.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_2.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_3.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_4.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_5.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_6.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_7.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_8.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_9.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_10.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_11.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_12.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_13.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_14.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_15.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_16.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_17.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_18.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_19.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_20.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_21.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_22.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_23.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_24.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_25.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_26.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_27.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_28.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_29.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_30.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_31.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_32.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_33.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_34.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_35.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_36.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_37.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_38.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_39.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_40.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_41.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_42.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_43.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_44.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_45.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_46.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_47.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_48.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_49.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_50.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_51.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_52.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_53.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_54.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_55.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_56.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_57.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_58.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_59.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_60.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_61.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_62.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_63.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_64.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_65.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_66.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_67.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_68.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_69.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_70.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_71.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_72.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_73.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_74.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_75.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_76.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_77.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_78.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_79.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_80.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_81.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_82.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_83.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_84.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_85.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_86.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_87.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_88.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_89.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_90.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_91.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_92.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_93.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_94.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_95.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_96.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_97.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_98.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "PUT /root/data/generated/file_99.csv HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] DEBUG : http://172.16.2.8:9001 "GET /root?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1" 200 0 (connectionpool.py:547, _make_request) 2025-05-23 21:38:28 [ 684 ] INFO : Starting mock server s3_mock.py (mock_servers.py:18, start_mock_servers) 2025-05-23 21:38:28 [ 684 ] DEBUG : run container_id:roottests3cluster-gw0-resolver-1 detach:False nothrow:False cmd: ['bash', '-c', 'mkdir -p $(dirname s3_mock.py) && echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py'] (cluster.py:2045, exec_in_container) 2025-05-23 21:38:28 [ 684 ] DEBUG : Command:[docker exec roottests3cluster-gw0-resolver-1 bash -c mkdir -p $(dirname s3_mock.py) && echo aW1wb3J0IHN5cwoKZnJvbSBib3R0bGUgaW1wb3J0IHJlcXVlc3QsIHJlc3BvbnNlLCByb3V0ZSwgcnVuCgoKQHJvdXRlKCIvPF9idWNrZXQ+LzxfcGF0aDpwYXRoPiIpCmRlZiBzZXJ2ZXIoX2J1Y2tldCwgX3BhdGgpOgogICAgcmVzdWx0ID0gKAogICAgICAgIHJlcXVlc3QuaGVhZGVyc1siTXlDdXN0b21IZWFkZXIiXQogICAgICAgIGlmICJNeUN1c3RvbUhlYWRlciIgaW4gcmVxdWVzdC5oZWFkZXJzCiAgICAgICAgZWxzZSAidW5rbm93biIKICAgICkKICAgIHJlc3BvbnNlLmNvbnRlbnRfdHlwZSA9ICJ0ZXh0L3BsYWluIgogICAgcmVzcG9uc2Uuc2V0X2hlYWRlcigiQ29udGVudC1MZW5ndGgiLCBsZW4ocmVzdWx0KSkKICAgIHJldHVybiByZXN1bHQKCgpAcm91dGUoIi8iKQpkZWYgcGluZygpOgogICAgcmVzcG9uc2UuY29udGVudF90eXBlID0gInRleHQvcGxhaW4iCiAgICByZXNwb25zZS5zZXRfaGVhZGVyKCJDb250ZW50LUxlbmd0aCIsIDIpCiAgICByZXR1cm4gIk9LIgoKCnJ1bihob3N0PSIwLjAuMC4wIiwgcG9ydD1pbnQoc3lzLmFyZ3ZbMV0pKQo= | base64 --decode > s3_mock.py] (cluster.py:121, run_and_check) 2025-05-23 21:38:28 [ 684 ] DEBUG : run container_id:roottests3cluster-gw0-resolver-1 detach:True nothrow:False cmd: ['bash', '-c', 'python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log'] (cluster.py:2045, exec_in_container) 2025-05-23 21:38:28 [ 684 ] DEBUG : Command:[docker exec roottests3cluster-gw0-resolver-1 bash -c python3 s3_mock.py 8080 >/var/log/resolver/s3_mock.log 2>/var/log/resolver/s3_mock.err.log] (cluster.py:121, run_and_check) 2025-05-23 21:38:28 [ 684 ] DEBUG : run container_id:roottests3cluster-gw0-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2045, exec_in_container) 2025-05-23 21:38:28 [ 684 ] DEBUG : Command:[docker exec roottests3cluster-gw0-resolver-1 curl -s http://localhost:8080/] (cluster.py:121, run_and_check) 2025-05-23 21:38:28 [ 684 ] DEBUG : Exitcode:7 (cluster.py:149, run_and_check) 2025-05-23 21:38:29 [ 684 ] DEBUG : run container_id:roottests3cluster-gw0-resolver-1 detach:False nothrow:True cmd: ['curl', '-s', 'http://localhost:8080/'] (cluster.py:2045, exec_in_container) 2025-05-23 21:38:29 [ 684 ] DEBUG : Command:[docker exec roottests3cluster-gw0-resolver-1 curl -s http://localhost:8080/] (cluster.py:121, run_and_check) 2025-05-23 21:38:29 [ 684 ] DEBUG : Stdout:OK (cluster.py:145, run_and_check) 2025-05-23 21:38:29 [ 684 ] DEBUG : s3_mock.py answered OK on attempt 2 (mock_servers.py:67, start_mock_servers) 2025-05-23 21:38:29 [ 684 ] INFO : Mock server s3_mock.py started (mock_servers.py:82, start_mock_servers) ----------------------------- Captured stderr call ----------------------------- Executing query insert into function s3('http://minio1:9001/root/data/data1', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select 1 as id settings s3_truncate_on_insert=1 on s0_0_0 Executing query insert into function s3('http://minio1:9001/root/data/data2', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select * from numbers(0) settings s3_truncate_on_insert=1 on s0_0_0 Executing query insert into function s3('http://minio1:9001/root/data/data3', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select 2 as id settings s3_truncate_on_insert=1 on s0_0_0 Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'auto', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'auto', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, test_s3_with_default) order by id on s0_0_0 Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:29 [ 684 ] DEBUG : Executing query insert into function s3('http://minio1:9001/root/data/data1', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select 1 as id settings s3_truncate_on_insert=1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:29 [ 684 ] DEBUG : Executing query insert into function s3('http://minio1:9001/root/data/data2', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select * from numbers(0) settings s3_truncate_on_insert=1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:29 [ 684 ] DEBUG : Executing query insert into function s3('http://minio1:9001/root/data/data3', 'minio', 'ClickHouse_Minio_P@ssw0rd', JSONEachRow) select 2 as id settings s3_truncate_on_insert=1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:29 [ 684 ] DEBUG : Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:30 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:30 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'auto', 'id UInt32, date Date DEFAULT 18262') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:30 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:30 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'auto', 'id UInt32, date Date DEFAULT 18262', 'auto') order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:30 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, test_s3_with_default) order by id on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:30 [ 684 ] DEBUG : Executing query SELECT * FROM s3('http://minio1:9001/root/data/data{1,2,3}', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'JSONEachRow', 'id UInt32, date Date DEFAULT 18262') order by id SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) ________________________ test_cluster_format_detection _________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_format_detection(started_cluster): node = started_cluster.instances["s0_0_0"] expected_desc_result = node.query( f"desc s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}', 'CSV')" ) desc_result = node.query( f"desc s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}')" ) assert expected_desc_result == desc_result expected_result = node.query( f"SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}', 'CSV', 'a String, b UInt64') order by a, b" ) result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2" ) assert result == expected_result result = node.query( f"SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}', auto, 'a String, b UInt64') order by a, b" ) assert result == expected_result > result = node.query( """SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:573: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query desc s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') on s0_0_0 Executing query desc s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd') on s0_0_0 Executing query SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'a String, b UInt64') order by a, b on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd') order by c1, c2 on s0_0_0 Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', auto, 'a String, b UInt64') order by a, b on s0_0_0 Executing query SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:30 [ 684 ] DEBUG : Executing query desc s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:31 [ 684 ] DEBUG : Executing query desc s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:31 [ 684 ] DEBUG : Executing query SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'a String, b UInt64') order by a, b on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:31 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd') order by c1, c2 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:31 [ 684 ] DEBUG : Executing query SELECT * FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/generated/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', auto, 'a String, b UInt64') order by a, b on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:31 [ 684 ] DEBUG : Executing query SELECT * FROM s3('http://minio1:9001/root/data/generated/*', 'minio', '{minio_secret_key}') order by c1, c2 SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) ___________________________ test_cluster_hosts_limit ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_hosts_limit(started_cluster): node = started_cluster.instances["s0_0_0"] query_id_def = str(uuid.uuid4()) resp_def = node.query( f""" SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) """, query_id=query_id_def, ) # object_storage_max_nodes is greater than number of hosts in cluster query_id_4_hosts = str(uuid.uuid4()) > resp_4_hosts = node.query( f""" SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_max_nodes=4 """, query_id=query_id_4_hosts, ) test_s3_cluster/test.py:1007: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_max_nodes is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT * from s3Cluster( E 'cluster_simple', E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) E SETTINGS object_storage_max_nodes=4 E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_max_nodes=4 on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:31 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:31 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_max_nodes=4 on s0_0_0 (cluster.py:3570, query) ___________________________ test_cluster_with_header ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_with_header(started_cluster): node = started_cluster.instances["s0_0_0"] assert ( node.query( "SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'))" ) == "SomeValue\n" ) assert ( node.query( "SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV')" ) == "SomeValue\n" ) assert ( node.query( "SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'))" ) == "SomeValue\n" ) assert ( node.query( "SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV')" ) == "SomeValue\n" ) > assert ( node.query( """SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) SETTINGS object_storage_cluster = 'cluster_simple'""" ) == "SomeValue\n" ) test_s3_cluster/test.py:496: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:32 [ 684 ] DEBUG : Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:32 [ 684 ] DEBUG : Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:32 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:32 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster('cluster_simple', 'http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue'), 'CSV') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:32 [ 684 ] DEBUG : Executing query SELECT * from s3('http://resolver:8080/bucket/key.csv', headers(MyCustomHeader = 'SomeValue')) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) ______________________ test_cluster_with_named_collection ______________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_cluster_with_named_collection(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query("""SELECT * from s3(test_s3) ORDER BY (c1, c2, c3)""") s3_cluster = node.query( """SELECT * from s3Cluster(cluster_simple, test_s3) ORDER BY (c1, c2, c3)""" ) assert TSV(pure_s3) == TSV(s3_cluster) s3_cluster = node.query( """SELECT * from s3Cluster(cluster_simple, test_s3, structure='auto') ORDER BY (c1, c2, c3)""" ) assert TSV(pure_s3) == TSV(s3_cluster) > s3_cluster = node.query( """SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:529: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) on s0_0_0 Executing query SELECT * from s3Cluster(cluster_simple, test_s3) ORDER BY (c1, c2, c3) on s0_0_0 Executing query SELECT * from s3Cluster(cluster_simple, test_s3, structure='auto') ORDER BY (c1, c2, c3) on s0_0_0 Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:32 [ 684 ] DEBUG : Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:32 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster(cluster_simple, test_s3) ORDER BY (c1, c2, c3) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:33 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster(cluster_simple, test_s3, structure='auto') ORDER BY (c1, c2, c3) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:33 [ 684 ] DEBUG : Executing query SELECT * from s3(test_s3) ORDER BY (c1, c2, c3) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) __________________________________ test_count __________________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_count(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')""" ) # print(pure_s3) s3_distributed = node.query( f""" SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')""" ) # print(s3_distributed) s3_distributed_alt_syntax = node.query( f""" SELECT count(*) from s3( > 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_simple'""" ) E NameError: name 'clickhouse' is not defined test_s3_cluster/test.py:161: NameError ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:33 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:33 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) _______________________________ test_count_macro _______________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_count_macro(started_cluster): node = started_cluster.instances["s0_0_0"] s3_macro = node.query( f""" SELECT count(*) from s3Cluster( '{{default_cluster_macro}}', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')""" ) # print(s3_distributed) s3_distributed = node.query( f""" SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')""" ) # print(s3_distributed) > s3_distributed_alt_syntax = node.query( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:190: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT count(*) from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', E 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( '{default_cluster_macro}', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:33 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( '{default_cluster_macro}', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:33 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:33 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) _______________________ test_distributed_s3_table_engine _______________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_distributed_s3_table_engine(started_cluster): node = started_cluster.instances["s0_0_0"] resp_def = node.query( f""" SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) """ ) node.query("DROP TABLE IF EXISTS single_node"); node.query( f""" CREATE TABLE single_node (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV') """ ) query_id_engine_single_node = str(uuid.uuid4()) resp_engine_single_node = node.query( """ SELECT * FROM single_node ORDER BY (name, value, polygon) """, query_id = query_id_engine_single_node ) assert resp_def == resp_engine_single_node node.query("DROP TABLE IF EXISTS distributed"); > node.query( f""" CREATE TABLE distributed (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV') SETTINGS object_storage_cluster='cluster_simple' """ ) test_s3_cluster/test.py:954: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Unknown setting 'object_storage_cluster'. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x0000000009f80b8b E 3. DB::BaseSettingsHelpers::throwSettingNotFound(std::basic_string_view>) @ 0x000000001259bd63 E 4. DB::BaseSettings::set(std::basic_string_view>, DB::Field const&) @ 0x0000000011e0b6d4 E 5. DB::(anonymous namespace)::createStorageObjectStorage(DB::StorageFactory::Arguments const&, std::shared_ptr) @ 0x000000001430239e E 6. std::shared_ptr std::__function::__policy_invoker (DB::StorageFactory::Arguments const&)>::__call_impl[abi:ne190107] (DB::StorageFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::StorageFactory::Arguments const&) @ 0x0000000014302f01 E 7. DB::StorageFactory::get(DB::ASTCreateQuery const&, String const&, std::shared_ptr, std::shared_ptr, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, DB::LoadingStrictnessLevel) const @ 0x0000000013f81b9f E 8. DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr>&, DB::LoadingStrictnessLevel) @ 0x000000001353c557 E 9. DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x00000000135311c2 E 10. DB::InterpreterCreateQuery::execute() @ 0x00000000135446d8 E 11. DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000001396de0b E 12. DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000013968824 E 13. DB::TCPHandler::runImpl() @ 0x0000000014c9d90c E 14. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 15. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 16. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 17. Poco::PooledThread::run() @ 0x00000000183b975b E 18. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 19. ? @ 0x00007f76b2f50ac3 E 20. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: CREATE TABLE distributed E (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) E ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') E SETTINGS object_storage_cluster='cluster_simple' E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query DROP TABLE IF EXISTS single_node on s0_0_0 Executing query CREATE TABLE single_node (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') on s0_0_0 Executing query SELECT * FROM single_node ORDER BY (name, value, polygon) on s0_0_0 Executing query DROP TABLE IF EXISTS distributed on s0_0_0 Executing query CREATE TABLE distributed (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') SETTINGS object_storage_cluster='cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query DROP TABLE IF EXISTS single_node on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query CREATE TABLE single_node (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query SELECT * FROM single_node ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query DROP TABLE IF EXISTS distributed on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query CREATE TABLE distributed (name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))) ENGINE=S3('http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV') SETTINGS object_storage_cluster='cluster_simple' on s0_0_0 (cluster.py:3570, query) __________________________ test_hive_partitioning[0] ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = allow_experimental_analyzer = 0 @pytest.mark.parametrize("allow_experimental_analyzer", [0, 1]) def test_hive_partitioning(started_cluster, allow_experimental_analyzer): node = started_cluster.instances["s0_0_0"] node.query(f"SET allow_experimental_analyzer = {allow_experimental_analyzer}") for i in range(1, 5): exists = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key={i}/*', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV """ ) if int(exists) == 0: node.query( f""" INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key={i}/data.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') SELECT {i}, {i} SETTINGS use_hive_partitioning = 0 """ ) query_id_full = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 """, query_id=query_id_full, ) result = int(result) assert result == 2 query_id_optimized = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 """, query_id=query_id_optimized, ) result = int(result) assert result == 2 query_id_cluster_full = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 """, query_id=query_id_cluster_full, ) result = int(result) assert result == 2 query_id_cluster_optimized = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 """, query_id=query_id_cluster_optimized, ) result = int(result) assert result == 2 node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple'") full_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_full}' FORMAT TSV """ ) full_traffic = int(full_traffic) assert full_traffic > 0 # 612*4 optimized_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_optimized}' FORMAT TSV """ ) optimized_traffic = int(optimized_traffic) assert optimized_traffic > 0 # 612*2 assert full_traffic > optimized_traffic cluster_full_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_cluster_full}' FORMAT TSV """ ) cluster_full_traffic = int(cluster_full_traffic) assert cluster_full_traffic == full_traffic cluster_optimized_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_cluster_optimized}' FORMAT TSV """ ) cluster_optimized_traffic = int(cluster_optimized_traffic) > assert cluster_optimized_traffic == optimized_traffic E assert 2984 == 1492 test_s3_cluster/test.py:919: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SET allow_experimental_analyzer = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=1/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 1, 1 SETTINGS use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=2/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=2/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 2, 2 SETTINGS use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=3/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=3/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 3, 3 SETTINGS use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=4/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=4/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 4, 4 SETTINGS use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple' on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='9b17dee2-b2ba-43a7-a366-6f18cbb68bfb' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='d0fa802c-ba0e-4fd3-a9ba-33348d390410' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='0a7681d2-fd85-406e-9b02-ea37a286df2c' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='085e1ca9-b996-4883-8f13-b863ead2b360' FORMAT TSV on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query SET allow_experimental_analyzer = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=1/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 1, 1 SETTINGS use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:34 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=2/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=2/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 2, 2 SETTINGS use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=3/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=3/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 3, 3 SETTINGS use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=4/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key=4/data.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') SELECT 4, 4 SETTINGS use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:35 [ 684 ] DEBUG : Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple' on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='9b17dee2-b2ba-43a7-a366-6f18cbb68bfb' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='d0fa802c-ba0e-4fd3-a9ba-33348d390410' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='0a7681d2-fd85-406e-9b02-ea37a286df2c' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='085e1ca9-b996-4883-8f13-b863ead2b360' FORMAT TSV on s0_0_0 (cluster.py:3570, query) __________________________ test_hive_partitioning[1] ___________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = allow_experimental_analyzer = 1 @pytest.mark.parametrize("allow_experimental_analyzer", [0, 1]) def test_hive_partitioning(started_cluster, allow_experimental_analyzer): node = started_cluster.instances["s0_0_0"] node.query(f"SET allow_experimental_analyzer = {allow_experimental_analyzer}") for i in range(1, 5): exists = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key={i}/*', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV """ ) if int(exists) == 0: node.query( f""" INSERT INTO FUNCTION s3('http://minio1:9001/root/data/hive/key={i}/data.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') SELECT {i}, {i} SETTINGS use_hive_partitioning = 0 """ ) query_id_full = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 """, query_id=query_id_full, ) result = int(result) assert result == 2 query_id_optimized = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 """, query_id=query_id_optimized, ) result = int(result) assert result == 2 query_id_cluster_full = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 """, query_id=query_id_cluster_full, ) result = int(result) assert result == 2 query_id_cluster_optimized = str(uuid.uuid4()) result = node.query( f""" SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', '{minio_secret_key}', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 """, query_id=query_id_cluster_optimized, ) result = int(result) assert result == 2 node.query("SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple'") full_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_full}' FORMAT TSV """ ) full_traffic = int(full_traffic) assert full_traffic > 0 # 612*4 optimized_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_optimized}' FORMAT TSV """ ) optimized_traffic = int(optimized_traffic) assert optimized_traffic > 0 # 612*2 assert full_traffic > optimized_traffic cluster_full_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_cluster_full}' FORMAT TSV """ ) cluster_full_traffic = int(cluster_full_traffic) assert cluster_full_traffic == full_traffic cluster_optimized_traffic = node.query( f""" SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='{query_id_cluster_optimized}' FORMAT TSV """ ) cluster_optimized_traffic = int(cluster_optimized_traffic) > assert cluster_optimized_traffic == optimized_traffic E assert 2984 == 1492 test_s3_cluster/test.py:919: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SET allow_experimental_analyzer = 1 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=2/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=3/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=4/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple' on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='db04fb78-6c90-4b6d-8862-988d7167ccb6' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='1c3927c2-b519-442a-987d-4dca3e24bda3' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='c793ee5b-aab4-4f68-a63f-ba84aad26c96' FORMAT TSV on s0_0_0 Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='331c90e2-093e-4343-af5d-a2d4d257284c' FORMAT TSV on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SET allow_experimental_analyzer = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=2/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=3/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=4/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 0 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:36 [ 684 ] DEBUG : Executing query SELECT count() FROM s3Cluster(cluster_simple, 'http://minio1:9001/root/data/hive/key=**.parquet', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'Parquet', 'key Int32, value Int32') WHERE key <= 2 FORMAT TSV SETTINGS enable_filesystem_cache = 0, use_query_cache = 0, use_cache_for_count_from_files = 0, use_hive_partitioning = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:37 [ 684 ] DEBUG : Executing query SYSTEM FLUSH LOGS ON CLUSTER 'cluster_simple' on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:37 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='db04fb78-6c90-4b6d-8862-988d7167ccb6' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:37 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='1c3927c2-b519-442a-987d-4dca3e24bda3' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:37 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='c793ee5b-aab4-4f68-a63f-ba84aad26c96' FORMAT TSV on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:37 [ 684 ] DEBUG : Executing query SELECT sum(ProfileEvents['ReadBufferFromS3Bytes']) FROM clusterAllReplicas(cluster_simple, system.query_log) WHERE type='QueryFinish' AND initial_query_id='331c90e2-093e-4343-af5d-a2d4d257284c' FORMAT TSV on s0_0_0 (cluster.py:3570, query) ______________________________ test_remote_hedged ______________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_remote_hedged(started_cluster): node = started_cluster.instances["s0_0_0"] > pure_s3 = node.query( f""" SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', 'minio{minio_secret_key}123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 """ ) test_s3_cluster/test.py:673: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 243, stderr: Received exception from server (version 25.3.3): E Code: 499. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Could not list objects in bucket 'root' with prefix 'data/', S3 exception: `SignatureDoesNotMatch`, message: 'The request signature we calculated does not match the signature you provided. Check your key and signing method.'. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::S3Exception::S3Exception(Aws::S3::S3Errors, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type>, String&&, String&&, String&&, String&&) @ 0x0000000011fea552 E 3. DB::(anonymous namespace)::S3IteratorAsync::getBatchAndCheckNext(std::vector, std::allocator>>&) @ 0x0000000011fea372 E 4. DB::IObjectStorageIteratorAsync::BatchAndHasNext std::__function::__policy_invoker::__call_impl[abi:ne190107]>(std::__function::__policy_storage const*) @ 0x000000001202c061 E 5. std::__packaged_task_func (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'(), std::allocator (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda'()>, DB::IObjectStorageIteratorAsync::BatchAndHasNext ()>::operator()() @ 0x000000001202b86d E 6. void std::__function::__policy_invoker::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority)::operator()(std::function&&, Priority)::'lambda0'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000001202bb40 E 7. ThreadPoolImpl>::ThreadFromThreadPool::worker() @ 0x000000000f5eba9b E 8. void std::__function::__policy_invoker::__call_impl[abi:ne190107]::ThreadFromGlobalPoolImpl>::ThreadFromThreadPool::*)(), ThreadPoolImpl>::ThreadFromThreadPool*>(void (ThreadPoolImpl>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl>::ThreadFromThreadPool*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000f5f25c2 E 9. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000000f5e8dcf E 10. void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000000f5f009a E 11. ? @ 0x00007f76b2f50ac3 E 12. ? @ 0x00007f76b2fe2850 E E Job's origin stack trace: E 0. void boost::heap::priority_queue, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function&&, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000000f5f0cd7 E 1. void ThreadPoolImpl>::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000000f5ece78 E 2. std::future std::__function::__policy_invoker (std::function&&, Priority)>::__call_impl[abi:ne190107] (std::function&&, Priority)> DB::threadPoolCallbackRunnerUnsafe>(ThreadPoolImpl>&, String const&)::'lambda'(std::function&&, Priority), std::future (std::function&&, Priority)>>(std::__function::__policy_storage const*, std::function&&, Priority&&) @ 0x000000001202af05 E 3. DB::IObjectStorageIteratorAsync::nextBatch() @ 0x0000000012029df0 E 4. DB::IObjectStorageIteratorAsync::isValid() @ 0x000000001202a5e2 E 5. DB::StorageObjectStorageSource::GlobIterator::estimatedKeysCount() @ 0x0000000011e6268a E 6. DB::(anonymous namespace)::ReadFromObjectStorageStep::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) (.764af0d3dc5e7addc6ed9b7904a05578) @ 0x0000000011e04bd9 E 7. DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x00000000151822ad E 8. DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x00000000151b3e4e E 9. DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000135c4e71 E 10. DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000135c479e E 11. DB::executeQueryImpl(char const*, char const*, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr&) @ 0x000000001396de0b E 12. DB::executeQuery(String const&, std::shared_ptr, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000013968824 E 13. DB::TCPHandler::runImpl() @ 0x0000000014c9d90c E 14. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 15. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 16. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 17. Poco::PooledThread::run() @ 0x00000000183b975b E 18. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 19. ? @ 0x00007f76b2f50ac3 E 20. ? @ 0x00007f76b2fe2850 E . (S3_ERROR) E (query: SELECT * from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', E 'minio', 'minioClickHouse_Minio_P@ssw0rd123', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E ORDER BY (name, value, polygon) E LIMIT 1 E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minioClickHouse_Minio_P@ssw0rd123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:37 [ 684 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minioClickHouse_Minio_P@ssw0rd123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 (cluster.py:3570, query) ____________________________ test_remote_no_hedged _____________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_remote_no_hedged(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query( f""" SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 """ ) > s3_distributed = node.query( f""" SELECT * from remote('s0_0_1', s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) ORDER BY (name, value, polygon) LIMIT 1 SETTINGS use_hedged_requests=False """ ) test_s3_cluster/test.py:710: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 49, stderr: Received exception from server (version 25.3.3): E Code: 49. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Distributed task iterator is not initialized: While executing Remote. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009f8d62b E 3. DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x0000000012840784 E 4. DB::RemoteQueryExecutor::readAsync() @ 0x0000000012841cb3 E 5. DB::RemoteSource::tryGenerate() @ 0x00000000150b06b0 E 6. DB::ISource::work() @ 0x0000000014d3cc87 E 7. DB::ExecutionThreadContext::executeTask() @ 0x0000000014d58876 E 8. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000014d4c364 E 9. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000014d4b53e E 10. void std::__function::__policy_invoker::__call_impl[abi:ne190107]::ThreadFromGlobalPoolImpl(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000014d5c011 E 11. ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000000f5e8dcf E 12. void* std::__thread_proxy[abi:ne190107]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000000f5f009a E 13. ? @ 0x00007f76b2f50ac3 E 14. ? @ 0x00007f76b2fe2850 E E Job's origin stack trace: E 0. void boost::heap::priority_queue, boost::parameter::void_, boost::parameter::void_, boost::parameter::void_>::emplace, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const, bool&, (anonymous namespace)::ScopedDecrement>(std::function&&, Priority&, StrongTypedef&, DB::OpenTelemetry::TracingContextOnThread const&&, bool&, (anonymous namespace)::ScopedDecrement&&) @ 0x000000000f5f0cd7 E 1. void ThreadPoolImpl::scheduleImpl(std::function, Priority, std::optional, bool) @ 0x000000000f5ea0b8 E 2. DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long) @ 0x0000000014d5ad53 E 3. DB::PullingAsyncPipelineExecutor::pull(DB::Block&, unsigned long) @ 0x0000000014d5b299 E 4. DB::TCPHandler::runImpl() @ 0x0000000014c9dd34 E 5. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 6. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 7. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 8. Poco::PooledThread::run() @ 0x00000000183b975b E 9. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 10. ? @ 0x00007f76b2f50ac3 E 11. ? @ 0x00007f76b2fe2850 E . (LOGICAL_ERROR) E (query: SELECT * from remote('s0_0_1', s3Cluster( E 'cluster_simple', E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) E ORDER BY (name, value, polygon) E LIMIT 1 E SETTINGS use_hedged_requests=False E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 Executing query SELECT * from remote('s0_0_1', s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) ORDER BY (name, value, polygon) LIMIT 1 SETTINGS use_hedged_requests=False on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:37 [ 684 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:38 [ 684 ] DEBUG : Executing query SELECT * from remote('s0_0_1', s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')) ORDER BY (name, value, polygon) LIMIT 1 SETTINGS use_hedged_requests=False on s0_0_0 (cluster.py:3570, query) _______________________________ test_select_all ________________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_select_all(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query( f""" SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon)""" ) # print(pure_s3) s3_distributed = node.query( f""" SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon)""" ) # print(s3_distributed) > s3_distributed_alt_syntax = node.query( f""" SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple'""" ) test_s3_cluster/test.py:128: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT * from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) E SETTINGS object_storage_cluster = 'cluster_simple') helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:38 [ 684 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:38 [ 684 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:38 [ 684 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) _________________________ test_skip_unavailable_shards _________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_skip_unavailable_shards(started_cluster): node = started_cluster.instances["s0_0_0"] result = node.query( f""" SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 """ ) assert result == "10\n" > result = node.query( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' """ ) test_s3_cluster/test.py:328: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT count(*) from s3( E 'http://minio1:9001/root/data/clickhouse/part1.csv', E 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 on s0_0_0 Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:38 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:38 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1, object_storage_cluster = 'cluster_non_existent_port' on s0_0_0 (cluster.py:3570, query) ________________________________ test_union_all ________________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_union_all(started_cluster): node = started_cluster.instances["s0_0_0"] pure_s3 = node.query( f""" SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) """ ) # print(pure_s3) s3_distributed = node.query( f""" SELECT * FROM ( SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) """ ) # print(s3_distributed) > s3_distributed_alt_syntax = node.query( f""" SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' """ ) test_s3_cluster/test.py:241: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT * FROM E ( E SELECT * from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E UNION ALL E SELECT * from s3( E 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', E 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E ) E ORDER BY (name, value, polygon) E SETTINGS object_storage_cluster = 'cluster_simple' E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * FROM ( SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:38 [ 684 ] DEBUG : Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:39 [ 684 ] DEBUG : Executing query SELECT * FROM ( SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:39 [ 684 ] DEBUG : Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) SETTINGS object_storage_cluster = 'cluster_simple' on s0_0_0 (cluster.py:3570, query) ______________________ test_unset_skip_unavailable_shards ______________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_unset_skip_unavailable_shards(started_cluster): # Although skip_unavailable_shards is not set, cluster table functions should always skip unavailable shards. node = started_cluster.instances["s0_0_0"] result = node.query( f""" SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') """ ) assert result == "10\n" > result = node.query( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_non_existent_port' """ ) test_s3_cluster/test.py:354: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3571: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 115, stderr: Received exception from server (version 25.3.3): E Code: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace: E E 0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb E 1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b E 3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8 E 4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944 E 5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3 E 6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4 E 7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303 E 8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4 E 9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9 E 10. DB::TCPHandler::run() @ 0x0000000014cbbaf9 E 11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7 E 12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419 E 13. Poco::PooledThread::run() @ 0x00000000183b975b E 14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d E 15. ? @ 0x00007f76b2f50ac3 E 16. ? @ 0x00007f76b2fe2850 E . (UNKNOWN_SETTING) E (query: SELECT count(*) from s3( E 'http://minio1:9001/root/data/clickhouse/part1.csv', E 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') E SETTINGS object_storage_cluster = 'cluster_non_existent_port' E ) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_non_existent_port' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:39 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3570, query) 2025-05-23 21:38:39 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'cluster_non_existent_port' on s0_0_0 (cluster.py:3570, query) ______________________________ test_wrong_cluster ______________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_wrong_cluster(started_cluster): node = started_cluster.instances["s0_0_0"] error = node.query_and_get_error( f""" SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') """ ) assert "not found" in error error = node.query_and_get_error( f""" SELECT count(*) from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3( 'http://minio1:9001/root/data/{{clickhouse,database}}/*', 'minio', '{minio_secret_key}', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'non_existing_cluster' """ ) > assert "not found" in error E assert 'not found' in "Received exception from server (version 25.3.3):\nCode: 115. DB::Exception: Received from 172.16.2.10:9000. DB::Exception: Setting object_storage_cluster is neither a builtin setting nor started with the prefix 'custom_' registered for user-defined settings. Stack trace:\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4b58fb\n1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009f80e6c\n2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x0000000009f8082b\n3. DB::AccessControl::CustomSettingsPrefixes::checkSettingNameIsAllowed(std::basic_string_view>) const @ 0x000000001232e9f8\n4. DB::SettingsConstraints::checkImpl(DB::Settings const&, DB::SettingChange&, DB::SettingsConstraints::ReactionOnViolation, DB::SettingSource) const @ 0x0000000012449944\n5. DB::SettingsConstraints::check(DB::Settings const&, DB::SettingsChanges&, DB::SettingSource) const @ 0x0000000012449cc3\n6. DB::Context::checkSettingsConstraintsWithLock(DB::SettingsChanges&, DB::SettingSource) @ 0x00000000130c30b4\n7. DB::TCPHandler::processQuery(std::optional&) @ 0x0000000014cb4303\n8. DB::TCPHandler::receivePacketsExpectQuery(std::optional&) @ 0x0000000014ca97f4\n9. DB::TCPHandler::runImpl() @ 0x0000000014c9cdb9\n10. DB::TCPHandler::run() @ 0x0000000014cbbaf9\n11. Poco::Net::TCPServerConnection::start() @ 0x00000000183ecfc7\n12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000183ed419\n13. Poco::PooledThread::run() @ 0x00000000183b975b\n14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000183b7c3d\n15. ? @ 0x00007f76b2f50ac3\n16. ? @ 0x00007f76b2fe2850\n. (UNKNOWN_SETTING)\n(query: SELECT count(*) from s3(\n 'http://minio1:9001/root/data/{clickhouse,database}/*',\n 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')\n UNION ALL\n SELECT count(*) from s3(\n 'http://minio1:9001/root/data/{clickhouse,database}/*',\n 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))')\n SETTINGS object_storage_cluster = 'non_existing_cluster'\n )\n" test_s3_cluster/test.py:293: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'non_existing_cluster' on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-05-23 21:38:39 [ 684 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3Cluster( 'non_existent_cluster', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') on s0_0_0 (cluster.py:3653, query_and_get_error) 2025-05-23 21:38:39 [ 684 ] DEBUG : Executing query SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT count(*) from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'ClickHouse_Minio_P@ssw0rd', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS object_storage_cluster = 'non_existing_cluster' on s0_0_0 (cluster.py:3653, query_and_get_error) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml stop --timeout 20] Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw0-resolver-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw0-zoo2-1 Stopping Stderr: Container roottests3cluster-gw0-minio1-1 Stopping Stderr: Container roottests3cluster-gw0-zoo1-1 Stopping Stderr: Container roottests3cluster-gw0-zoo3-1 Stopping Stderr: Container roottests3cluster-gw0-minio1-1 Stopped Stderr: Container roottests3cluster-gw0-zoo2-1 Stopped Stderr: Container roottests3cluster-gw0-zoo3-1 Stopped Stderr: Container roottests3cluster-gw0-zoo1-1 Stopped Stderr: Container roottests3cluster-gw0-resolver-1 Stopped Stderr: Container roottests3cluster-gw0-proxy1-1 Stopping Stderr: Container roottests3cluster-gw0-proxy2-1 Stopping Stderr: Container roottests3cluster-gw0-proxy2-1 Stopped Stderr: Container roottests3cluster-gw0-proxy1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml down --volumes] Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw0-resolver-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw0-s0_0_1-1 Removing Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw0-s0_1_0-1 Removing Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw0-s0_0_0-1 Removing Stderr: Container roottests3cluster-gw0-resolver-1 Stopped Stderr: Container roottests3cluster-gw0-resolver-1 Removing Stderr: Container roottests3cluster-gw0-s0_0_0-1 Removed Stderr: Container roottests3cluster-gw0-minio1-1 Stopping Stderr: Container roottests3cluster-gw0-s0_0_1-1 Removed Stderr: Container roottests3cluster-gw0-minio1-1 Stopped Stderr: Container roottests3cluster-gw0-minio1-1 Removing Stderr: Container roottests3cluster-gw0-s0_1_0-1 Removed Stderr: Container roottests3cluster-gw0-resolver-1 Removed Stderr: Container roottests3cluster-gw0-zoo1-1 Stopping Stderr: Container roottests3cluster-gw0-zoo2-1 Stopping Stderr: Container roottests3cluster-gw0-zoo3-1 Stopping Stderr: Container roottests3cluster-gw0-zoo2-1 Stopped Stderr: Container roottests3cluster-gw0-zoo2-1 Removing Stderr: Container roottests3cluster-gw0-zoo3-1 Stopped Stderr: Container roottests3cluster-gw0-zoo3-1 Removing Stderr: Container roottests3cluster-gw0-zoo1-1 Stopped Stderr: Container roottests3cluster-gw0-zoo1-1 Removing Stderr: Container roottests3cluster-gw0-zoo1-1 Removed Stderr: Container roottests3cluster-gw0-zoo3-1 Removed Stderr: Container roottests3cluster-gw0-zoo2-1 Removed Stderr: Container roottests3cluster-gw0-minio1-1 Removed Stderr: Container roottests3cluster-gw0-proxy1-1 Stopping Stderr: Container roottests3cluster-gw0-proxy2-1 Stopping Stderr: Container roottests3cluster-gw0-proxy1-1 Stopped Stderr: Container roottests3cluster-gw0-proxy1-1 Removing Stderr: Container roottests3cluster-gw0-proxy2-1 Stopped Stderr: Container roottests3cluster-gw0-proxy2-1 Removing Stderr: Container roottests3cluster-gw0-proxy2-1 Removed Stderr: Container roottests3cluster-gw0-proxy1-1 Removed Stderr: Volume roottests3cluster-gw0_data1-1 Removing Stderr: Network roottests3cluster-gw0_default Removing Stderr: Volume roottests3cluster-gw0_data1-1 Removed Stderr: Network roottests3cluster-gw0_default Removed Cleanup called Docker networks for project roottests3cluster-gw0 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster-gw0 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3cluster-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottests3cluster-gw0 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 ---------------------------- Captured log teardown ----------------------------- 2025-05-23 21:38:39 [ 684 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml stop --timeout 20] (cluster.py:121, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/.env --project-name roottests3cluster-gw0 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-1-gw0/s0_1_0/docker-compose.yml down --volumes] (cluster.py:121, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_0-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_0_1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-s0_1_0-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-resolver-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo3-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-zoo2-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-minio1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy2-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Container roottests3cluster-gw0-proxy1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Volume roottests3cluster-gw0_data1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Network roottests3cluster-gw0_default Removing (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Volume roottests3cluster-gw0_data1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Stderr: Network roottests3cluster-gw0_default Removed (cluster.py:147, run_and_check) 2025-05-23 21:39:00 [ 684 ] DEBUG : Cleanup called (cluster.py:846, cleanup) 2025-05-23 21:39:00 [ 684 ] DEBUG : Docker networks for project roottests3cluster-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:825, print_all_docker_pieces) 2025-05-23 21:39:01 [ 684 ] DEBUG : Docker containers for project roottests3cluster-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:833, print_all_docker_pieces) 2025-05-23 21:39:01 [ 684 ] DEBUG : Docker volumes for project roottests3cluster-gw0 are DRIVER VOLUME NAME (cluster.py:841, print_all_docker_pieces) 2025-05-23 21:39:01 [ 684 ] DEBUG : Command:[docker container list --all --filter name='^/roottests3cluster-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2025-05-23 21:39:01 [ 684 ] DEBUG : Unstopped containers: {} (cluster.py:860, cleanup) 2025-05-23 21:39:01 [ 684 ] DEBUG : No running containers for project: roottests3cluster-gw0 (cluster.py:874, cleanup) 2025-05-23 21:39:01 [ 684 ] DEBUG : Trying to prune unused networks... (cluster.py:880, cleanup) 2025-05-23 21:39:01 [ 684 ] DEBUG : Trying to prune unused images... (cluster.py:896, cleanup) 2025-05-23 21:39:01 [ 684 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2025-05-23 21:39:01 [ 684 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2025-05-23 21:39:01 [ 684 ] DEBUG : Images pruned (cluster.py:899, cleanup) 2025-05-23 21:39:01 [ 684 ] DEBUG : Trying to prune unused volumes... (cluster.py:905, cleanup) 2025-05-23 21:39:01 [ 684 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2025-05-23 21:39:01 [ 684 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2025-05-23 21:39:01 [ 684 ] DEBUG : Volumes pruned: 1 (cluster.py:910, cleanup) ___________________________ test_update_one_cluster ____________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_update_one_cluster(started_cluster): send_repeated_query("distributed") send_repeated_query("distributed2") > assert get_errors_count("test_cluster") > 0 E AssertionError: assert 0 > 0 E + where 0 = get_errors_count('test_cluster') test_reload_clusters_config/test.py:202: AssertionError ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node ------------------------------ Captured log call ------------------------------- 2025-05-23 21:40:48 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:41:00 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:41:13 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:41:26 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:41:38 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:41:50 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:42:02 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:42:14 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:42:26 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:42:39 [ 687 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3653, query_and_get_error) 2025-05-23 21:42:51 [ 687 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node (cluster.py:3570, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/.env --project-name roottestreloadclustersconfig-gw1 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] Stderr: Container roottestreloadclustersconfig-gw1-node-1 Stopping Stderr: Container roottestreloadclustersconfig-gw1-node-1 Stopped Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Stopping Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Stopping Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Stopping Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Stopped Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Stopped Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/.env --project-name roottestreloadclustersconfig-gw1 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] Stderr: Container roottestreloadclustersconfig-gw1-node-1 Stopping Stderr: Container roottestreloadclustersconfig-gw1-node-1 Stopped Stderr: Container roottestreloadclustersconfig-gw1-node-1 Removing Stderr: Container roottestreloadclustersconfig-gw1-node-1 Removed Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Stopping Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Stopping Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Stopping Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Stopped Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Removing Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Stopped Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Removing Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Stopped Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Removing Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Removed Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Removed Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Removed Stderr: Network roottestreloadclustersconfig-gw1_default Removing Stderr: Network roottestreloadclustersconfig-gw1_default Removed Cleanup called Docker networks for project roottestreloadclustersconfig-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottestreloadclustersconfig-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottestreloadclustersconfig-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} No running containers for project: roottestreloadclustersconfig-gw1 Trying to prune unused networks... Trying to prune unused images... Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Images pruned Trying to prune unused volumes... Command:[docker volume ls | wc -l] Stdout:1 Volumes pruned: 1 ---------------------------- Captured log teardown ----------------------------- 2025-05-23 21:42:51 [ 687 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/.env --project-name roottestreloadclustersconfig-gw1 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] (cluster.py:121, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-node-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-node-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/.env --project-name roottestreloadclustersconfig-gw1 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw1/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] (cluster.py:121, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-node-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-node-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-node-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-node-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Removing (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo3-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo2-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw1-zoo1-1 Removed (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw1_default Removing (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw1_default Removed (cluster.py:147, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Cleanup called (cluster.py:846, cleanup) 2025-05-23 21:42:59 [ 687 ] DEBUG : Docker networks for project roottestreloadclustersconfig-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:825, print_all_docker_pieces) 2025-05-23 21:42:59 [ 687 ] DEBUG : Docker containers for project roottestreloadclustersconfig-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:833, print_all_docker_pieces) 2025-05-23 21:42:59 [ 687 ] DEBUG : Docker volumes for project roottestreloadclustersconfig-gw1 are DRIVER VOLUME NAME (cluster.py:841, print_all_docker_pieces) 2025-05-23 21:42:59 [ 687 ] DEBUG : Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Unstopped containers: {} (cluster.py:860, cleanup) 2025-05-23 21:42:59 [ 687 ] DEBUG : No running containers for project: roottestreloadclustersconfig-gw1 (cluster.py:874, cleanup) 2025-05-23 21:42:59 [ 687 ] DEBUG : Trying to prune unused networks... (cluster.py:880, cleanup) 2025-05-23 21:42:59 [ 687 ] DEBUG : Trying to prune unused images... (cluster.py:896, cleanup) 2025-05-23 21:42:59 [ 687 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Images pruned (cluster.py:899, cleanup) 2025-05-23 21:42:59 [ 687 ] DEBUG : Trying to prune unused volumes... (cluster.py:905, cleanup) 2025-05-23 21:42:59 [ 687 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2025-05-23 21:42:59 [ 687 ] DEBUG : Volumes pruned: 1 (cluster.py:910, cleanup) ----------------- generated report log file: parallel2_1.jsonl ----------------- ============================== slowest durations =============================== 134.61s call test_reload_clusters_config/test.py::test_delete_cluster 123.14s call test_reload_clusters_config/test.py::test_update_one_cluster 24.81s setup test_reload_clusters_config/test.py::test_delete_cluster 21.32s teardown test_s3_cluster/test.py::test_wrong_cluster 20.59s setup test_s3_cluster/test.py::test_cluster_default_expression 8.25s teardown test_reload_clusters_config/test.py::test_update_one_cluster 1.63s call test_s3_cluster/test.py::test_hive_partitioning[0] 1.36s call test_s3_cluster/test.py::test_hive_partitioning[1] 1.10s call test_s3_cluster/test.py::test_cluster_default_expression 0.79s call test_s3_cluster/test.py::test_cluster_format_detection 0.53s call test_s3_cluster/test.py::test_cluster_with_header 0.49s call test_s3_cluster/test.py::test_distributed_s3_table_engine 0.45s call test_s3_cluster/test.py::test_count_macro 0.42s call test_s3_cluster/test.py::test_cluster_with_named_collection 0.30s call test_s3_cluster/test.py::test_union_all 0.30s call test_s3_cluster/test.py::test_select_all 0.23s call test_s3_cluster/test.py::test_count 0.23s call test_s3_cluster/test.py::test_remote_no_hedged 0.18s call test_s3_cluster/test.py::test_cluster_hosts_limit 0.18s call test_s3_cluster/test.py::test_unset_skip_unavailable_shards 0.18s call test_s3_cluster/test.py::test_skip_unavailable_shards 0.13s call test_s3_cluster/test.py::test_wrong_cluster 0.07s call test_s3_cluster/test.py::test_remote_hedged 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[1] 0.00s setup test_s3_cluster/test.py::test_remote_hedged 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[0] 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression 0.00s teardown test_s3_cluster/test.py::test_count 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning[1] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header 0.00s teardown test_reload_clusters_config/test.py::test_delete_cluster 0.00s setup test_s3_cluster/test.py::test_count_macro 0.00s teardown test_s3_cluster/test.py::test_unset_skip_unavailable_shards 0.00s setup test_s3_cluster/test.py::test_hive_partitioning[0] 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards 0.00s setup test_s3_cluster/test.py::test_cluster_with_header 0.00s teardown test_s3_cluster/test.py::test_select_all 0.00s teardown test_s3_cluster/test.py::test_distributed_s3_table_engine 0.00s setup test_s3_cluster/test.py::test_union_all 0.00s setup test_reload_clusters_config/test.py::test_update_one_cluster 0.00s teardown test_s3_cluster/test.py::test_cluster_hosts_limit 0.00s teardown test_s3_cluster/test.py::test_remote_no_hedged 0.00s setup test_s3_cluster/test.py::test_unset_skip_unavailable_shards 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection 0.00s teardown test_s3_cluster/test.py::test_union_all 0.00s setup test_s3_cluster/test.py::test_wrong_cluster 0.00s teardown test_s3_cluster/test.py::test_count_macro 0.00s setup test_s3_cluster/test.py::test_distributed_s3_table_engine 0.00s setup test_s3_cluster/test.py::test_select_all 0.00s teardown test_s3_cluster/test.py::test_remote_hedged 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s setup test_s3_cluster/test.py::test_count 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection 0.00s setup test_s3_cluster/test.py::test_remote_no_hedged 0.00s setup test_s3_cluster/test.py::test_cluster_hosts_limit =========================== short test summary info ============================ FAILED test_s3_cluster/test.py::test_cluster_default_expression - helpers.cli... FAILED test_s3_cluster/test.py::test_cluster_format_detection - helpers.clien... FAILED test_s3_cluster/test.py::test_cluster_hosts_limit - helpers.client.Que... FAILED test_s3_cluster/test.py::test_cluster_with_header - helpers.client.Que... FAILED test_s3_cluster/test.py::test_cluster_with_named_collection - helpers.... FAILED test_s3_cluster/test.py::test_count - NameError: name 'clickhouse' is ... FAILED test_s3_cluster/test.py::test_count_macro - helpers.client.QueryRuntim... FAILED test_s3_cluster/test.py::test_distributed_s3_table_engine - helpers.cl... FAILED test_s3_cluster/test.py::test_hive_partitioning[0] - assert 2984 == 1492 FAILED test_s3_cluster/test.py::test_hive_partitioning[1] - assert 2984 == 1492 FAILED test_s3_cluster/test.py::test_remote_hedged - helpers.client.QueryRunt... FAILED test_s3_cluster/test.py::test_remote_no_hedged - helpers.client.QueryR... FAILED test_s3_cluster/test.py::test_select_all - helpers.client.QueryRuntime... FAILED test_s3_cluster/test.py::test_skip_unavailable_shards - helpers.client... FAILED test_s3_cluster/test.py::test_union_all - helpers.client.QueryRuntimeE... FAILED test_s3_cluster/test.py::test_unset_skip_unavailable_shards - helpers.... FAILED test_s3_cluster/test.py::test_wrong_cluster - assert 'not found' in "R... FAILED test_reload_clusters_config/test.py::test_update_one_cluster - Asserti... PASSED test_reload_clusters_config/test.py::test_delete_cluster =================== 18 failed, 1 passed in 292.87s (0:04:52) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_mumvsz --privileged --dns-search='.' --memory=30709035008 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel2_1.jsonl --report-log-exclude-logs-on-passed-tests test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_hosts_limit test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_s3_table_engine 'test_s3_cluster/test.py::test_hive_partitioning[0]' 'test_s3_cluster/test.py::test_hive_partitioning[1]' test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all test_s3_cluster/test.py::test_unset_skip_unavailable_shards test_s3_cluster/test.py::test_wrong_cluster -vvv " altinityinfra/integration-tests-runner:ad96270260ff ' returned non-zero exit status 1.